Sample autogenerated report

This section contains a CI sample report for running inference on a compiled model.

The CI is set up as follows:

  • Environment: Github Actions

  • Task: dog and cat breeds classification based on the Oxford-IIIT Pet Dataset,

  • Training framework: TensorFlow,

  • Compiler framework: TVM,

  • Target: CPU (using LLVM target on TVM).

Pet Dataset classification using TVM-compiled TensorFlow model

Commands used

Note

This section was generated using:

python -m kenning.scenarios.inference_tester \
    kenning.modelwrappers.classification.tensorflow_pet_dataset.TensorFlowPetDatasetMobileNetV2 \
    kenning.datasets.pet_dataset.PetDataset \
    ./build/local-cpu-tvm-tensorflow-classification.json \
    --compiler-cls \
        kenning.compilers.tvm.TVMCompiler \
    --runtime-cls \
        kenning.runtimes.tvm.TVMRuntime \
    --model-path \
        ./kenning/resources/models/classification/tensorflow_pet_dataset_mobilenetv2.h5 \
    --model-framework \
        keras \
    --target \
        llvm \
    --compiled-model-path \
        ./build/compiled-model.tar \
    --opt-level \
        3 \
    --save-model-path \
        ./build/compiled-model.tar \
    --target-device-context \
        cpu \
    --dataset-root \
        ./build/pet-dataset/ \
    --inference-batch-size \
        1 \
    --download-dataset \
    --verbosity \
        INFO

python -m kenning.scenarios.render_report \
    Pet Dataset classification using TVM-compiled TensorFlow model \
    docs/source/generated/local-cpu-tvm-tensorflow-classification.md \
    --root-dir \
        docs/source/ \
    --img-dir \
        docs/source/generated/img \
    --report-types \
        performance \
        classification \
    --measurements \
        build/local-cpu-tvm-tensorflow-classification.json

General information for build.local-cpu-tvm-tensorflow-classification.json

Model framework:

  • tensorflow ver. 2.9.3

Compiler framework:

  • tvm ver. 0.11.1

Input JSON:

{
    "dataset": {
        "type": "kenning.datasets.pet_dataset.PetDataset",
        "parameters": {
            "classify_by": "breeds",
            "image_memory_layout": "NHWC",
            "dataset_root": "build/pet-dataset",
            "inference_batch_size": 1,
            "download_dataset": true,
            "external_calibration_dataset": null
        }
    },
    "model_wrapper": {
        "type": "kenning.modelwrappers.classification.tensorflow_pet_dataset.TensorFlowPetDatasetMobileNetV2",
        "parameters": {
            "model_path": "kenning/resources/models/classification/tensorflow_pet_dataset_mobilenetv2.h5"
        }
    },
    "runtime": {
        "type": "kenning.runtimes.tvm.TVMRuntime",
        "parameters": {
            "save_model_path": "build/compiled-model.tar",
            "target_device_context": "cpu",
            "target_device_context_id": 0,
            "runtime_use_vm": false,
            "disable_performance_measurements": true
        }
    },
    "optimizers": [
        {
            "type": "kenning.compilers.tvm.TVMCompiler",
            "parameters": {
                "model_framework": "keras",
                "target": "llvm",
                "target_host": null,
                "opt_level": 3,
                "libdarknet_path": "/usr/local/lib/libdarknet.so",
                "compile_use_vm": false,
                "output_conversion_function": "default",
                "conv2d_data_layout": "",
                "conv2d_kernel_layout": "",
                "use_fp16_precision": false,
                "use_int8_precision": false,
                "use_tensorrt": false,
                "dataset_percentage": 0.25,
                "compiled_model_path": "build/compiled-model.tar"
            }
        }
    ]
}

Inference performance metrics for build.local-cpu-tvm-tensorflow-classification.json

Inference time

Bokeh Application

Figure 1 Inference time

  • First inference duration (usually including allocation time): 0.17529404300000806,

  • Mean: 0.16417127104653828 s,

  • Standard deviation: 0.002196815747795452 s,

  • Median: 0.16378020400009063 s.

Average CPU usage

Bokeh Application

Figure 2 Average CPU usage during benchmark

  • Mean: 51.14232200839471 %,

  • Standard deviation: 6.221318910822455 %,

  • Median: 50.0 %.

Memory usage

Bokeh Application

Figure 3 Memory usage during benchmark

  • Mean: 19.696697552862915 %,

  • Standard deviation: 0.19819257109373487 %,

  • Median: 19.7 %.

Inference quality metrics for build.local-cpu-tvm-tensorflow-classification.json

Bokeh Plot

Figure 4 Confusion matrix

  • Accuracy: 0.957273098380732

  • Top-5 accuracy: 0.9961899578173902

  • Mean precision: 0.9574413909275928

  • Mean sensitivity: 0.9571990915922841

  • G-mean: 0.9566448082813757

Last update: 2023-04-14